Attribute Information:¶
ATTRIBUTES¶
COMPACTNESS (average perim)**2/area
CIRCULARITY (average radius)**2/area
DISTANCE CIRCULARITY area/(av.distance from border)**2
RADIUS RATIO (max.rad-min.rad)/av.radius
PR.AXIS ASPECT RATIO (minor axis)/(major axis)
MAX.LENGTH ASPECT RATIO (length perp. max length)/(max length)
SCATTER RATIO (inertia about minor axis)/(inertia about major axis)
ELONGATEDNESS area/(shrink width)**2
PR.AXIS RECTANGULARITY area/(pr.axis length*pr.axis width)
MAX.LENGTH RECTANGULARITY area/(max.length*length perp. to this)
SCALED VARIANCE (2nd order moment about minor axis)/area ALONG MAJOR AXIS
SCALED VARIANCE.1 (2nd order moment about major axis)/area ALONG MINOR AXIS
SCALED RADIUS OF GYRATION (mavar+mivar)/area
SCALED RADIUS OF GYRATION.1
SKEWNESS ABOUT
SKEWNESS ABOUT1 (3rd order moment about major axis)/sigma_min**3 MAJOR AXIS
SKEWNESS ABOUT2 (3rd order moment about minor axis)/sigma_maj**3 MINOR AXIS
HOLLOWS RATIO (area of hollows)/(area of bounding polygon)
CLASS (CLASS OF VEHICLE: CAR, VAN, BUS)
In [191]:
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
from scipy.stats import zscore
from sklearn.model_selection import train_test_split
from sklearn import model_selection
from sklearn.model_selection import KFold
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
from sklearn import metrics
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.model_selection import cross_val_score
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
In [192]:
#Loading the dataset and storing in dataframe
vehicle_df = pd.read_csv('vehicle-2.csv')
In [193]:
vehicle_df.head()
Out[193]:
In [194]:
#Checking Width and Depth of the Data
vehicle_df.shape
Out[194]:
In [195]:
#Finding Datatypes of Each Attribute in the data
vehicle_df.dtypes
Out[195]:
In [196]:
#Analysing the Class Counts on basis of target variable: 'class' of vehicle
vehicle_df['class'].value_counts()
Out[196]:
In [197]:
#Checking and Analysing Object Type Data
vehicle_df.describe().transpose()
Out[197]:
In [198]:
#Checking for NUll Values in the dataframe..
vehicle_df.isnull().any()
Out[198]:
So, there are null values in the dataframe.¶
Following Attributes have Null Vlues in dataframe.
- circularity
- distance_circularity
- radius_ratio
- pr.axis_aspect_ratio
- scatter_ratio
- elongatedness
- pr.axis_rectangularity
- scaled_variance
- scaled_variance.1
- scaled_radius_of_gyration
- scaled_radius_of_gyration.1
- skewness_about
- skewness_about.1
- skewness_about.2
###### 14 Attributes in the dataset have null values
In [199]:
#Plotting Pie Chart and Count Plot for Term Deposit
figure,ax = plt.subplots(1,2,figsize=(15,7))
labels =['Car', 'Bus', 'Van']
ax[0].set_title("Deposit Distribution Pie")
vehicle_df['class'].value_counts().plot.pie(ax = ax[0], labels =['Car', 'Bus', 'Van'], autopct='%1.1f%%', colors = ['#176BB8', '#F6E708', '#ED1515'])
plt.title('Vehicle Class Index')
plt.legend(labels,loc=3)
sns.countplot(vehicle_df['class'], ax = ax[1], palette = ['#ED1515', '#176BB8', '#F6E708'])
ax[1].set_xticklabels(['Van', 'Car', 'Bus'])
vehicle_df['class'].value_counts()
Out[199]:
In [200]:
#Object type Data cannot be handled by Algorithms. Changing the target variable type to 'Category'.
#Mapping the yes and no values to '0' and '1'
vehicle_df['class'] = vehicle_df['class'].astype('category')
vehicle_df['class'] = vehicle_df['class'].replace(['car'], 1)
vehicle_df['class'] = vehicle_df['class'].replace(['bus'], 2)
vehicle_df['class'] = vehicle_df['class'].replace(['van'], 3)
vehicle_df['class'].value_counts()
Out[200]:
In [201]:
# Defining Features and Target:
vehiclef_df = vehicle_df.drop(["class"],axis=1)
class_df = vehicle_df["class"]
In [202]:
vehiclef_df.isnull().sum()
Out[202]:
In [203]:
!pip install missingno
import missingno as msno
msno.bar(vehiclef_df)
Out[203]:
14 columns out of 18 features have missing values. Total no. of value each column has is indicated at the top of each bar. We need to address these missing values
In [204]:
# Handling Nulls: Replacing Nulls by mean value
def remove_nulls(df_in, col_name):
df_in[col_name].fillna((df_in[col_name].mean()), inplace=True)
return df_in
clean_vehicle = remove_nulls(vehiclef_df, 'compactness')
clean_vehicle = remove_nulls(vehiclef_df, 'circularity')
clean_vehicle = remove_nulls(vehiclef_df, 'distance_circularity')
clean_vehicle = remove_nulls(vehiclef_df, 'radius_ratio')
clean_vehicle = remove_nulls(vehiclef_df, 'pr.axis_aspect_ratio')
clean_vehicle = remove_nulls(vehiclef_df, 'scatter_ratio')
clean_vehicle = remove_nulls(vehiclef_df, 'elongatedness')
clean_vehicle = remove_nulls(vehiclef_df, 'pr.axis_rectangularity')
clean_vehicle = remove_nulls(vehiclef_df, 'max.length_rectangularity')
clean_vehicle = remove_nulls(vehiclef_df, 'scaled_variance')
clean_vehicle = remove_nulls(vehiclef_df, 'scaled_variance.1')
clean_vehicle = remove_nulls(vehiclef_df, 'scaled_radius_of_gyration')
clean_vehicle = remove_nulls(vehiclef_df, 'scaled_radius_of_gyration.1')
clean_vehicle = remove_nulls(vehiclef_df, 'skewness_about')
clean_vehicle = remove_nulls(vehiclef_df, 'skewness_about.1')
clean_vehicle = remove_nulls(vehiclef_df, 'skewness_about.2')
clean_vehicle = remove_nulls(vehiclef_df, 'hollows_ratio')
In [205]:
clean_vehicle.isnull().sum().sum()
Out[205]:
Null Values treated
Handling Outliers: Plotting Coluns with Outliers.¶
In [206]:
# Plotting the Columns with Outliers
plt.figure(figsize = (15, 15))
boxplot = vehiclef_df.boxplot(column=['radius_ratio', 'pr.axis_aspect_ratio', 'max.length_aspect_ratio', 'scaled_variance', 'scaled_variance.1', 'scaled_radius_of_gyration.1', 'skewness_about'], grid = False)
In [235]:
# Handling Outliers using IQR method
def handle_outlier(df_in, col_name):
q1 = df_in[col_name].quantile(0.25)
q3 = df_in[col_name].quantile(0.75)
iqr = q3-q1 #Interquartile range
fence_low = q1-1.5*iqr
fence_high = q3+1.5*iqr
print('IQR lower bound and upper bound of', col_name, 'is', fence_low, 'and', fence_high, 'respectively')
df_in.loc[df_in[col_name] > fence_high, col_name] = df_in[col_name].median()
return df_in
clean_vehicle2 = handle_outlier(clean_vehicle, 'compactness')
clean_vehicle2 = handle_outlier(clean_vehicle, 'circularity')
clean_vehicle2 = handle_outlier(clean_vehicle, 'distance_circularity')
clean_vehicle2 = handle_outlier(clean_vehicle, 'radius_ratio')
clean_vehicle2 = handle_outlier(clean_vehicle, 'pr.axis_aspect_ratio')
clean_vehicle2 = handle_outlier(clean_vehicle, 'max.length_aspect_ratio')
clean_vehicle2 = handle_outlier(clean_vehicle, 'scatter_ratio')
clean_vehicle2 = handle_outlier(clean_vehicle, 'elongatedness')
clean_vehicle2 = handle_outlier(clean_vehicle, 'pr.axis_rectangularity')
clean_vehicle2 = handle_outlier(clean_vehicle, 'max.length_rectangularity')
clean_vehicle2 = handle_outlier(clean_vehicle, 'scaled_variance')
clean_vehicle2 = handle_outlier(clean_vehicle, 'scaled_variance.1')
clean_vehicle2 = handle_outlier(clean_vehicle, 'scaled_radius_of_gyration')
clean_vehicle2 = handle_outlier(clean_vehicle, 'scaled_radius_of_gyration.1')
clean_vehicle2 = handle_outlier(clean_vehicle, 'skewness_about')
clean_vehicle2 = handle_outlier(clean_vehicle, 'skewness_about.1')
clean_vehicle2 = handle_outlier(clean_vehicle, 'skewness_about.2')
clean_vehicle2 = handle_outlier(clean_vehicle, 'hollows_ratio')
print('outliers handled successfully')
In [208]:
clean_vehicle2.head()
Out[208]:
In [209]:
# Defining Features and Target:
feature = clean_vehicle2
target = vehicle_df["class"]
In [210]:
#Plotting Features HeatMap
corr = feature.corr()
mask = np.zeros_like(corr)
mask[np.triu_indices_from(mask)] = True
plt.figure(figsize = (12, 12))
sns.heatmap(corr, mask = mask,annot = True, square = True)
Out[210]:
circularity, distance_circularity, scatter_ratio, elongatedness, pr.axis_rectangularity and scaled_variance are the dimensions that are highly correlated to each other.¶
So we have a problem of multicollinearity here.¶
"multicollinearity" refers to predictors that are correlated with other predictors. Multicollinearity occurs when your model includes multiple factors that are correlated not just to our response variable, but also to each other.¶
It can reduce our overall coefficient as well as our p-value (known as the significance value) and cause unpredictable variance.¶
This will lead to overfitting where the model may do great on known training set but will fail at unknown testing set.¶
As this leads to higher standard error with lower statistical significance value, multicollinearity makes it difficult to ascertain how important a feature is to the target variable. And with a lower significance value, we will fail to reject the null, which leads to type II error.¶
In [211]:
feature_target_corr = feature.join(target).corr()
mask = np.zeros((19, 19))
mask[:18, :] = 1
plt.figure(figsize=(15, 8))
with sns.axes_style("white"):
sns.heatmap(feature_target_corr, annot=True, mask = mask)
Only elongatedness i s significantly correlated to 'class' of vehicle i.e. our target variable.
In [114]:
# concatenating features from cleaned dataframe and target column i.e. class from original dataframe
complete_data = clean_vehicle2
complete_data['class'] = vehicle_df['class']
complete_data.shape
Out[114]:
Plottting Distributions of Attributes:¶
In [118]:
col_names = ['compactness', 'circularity', 'distance_circularity', 'radius_ratio', 'pr.axis_aspect_ratio', 'max.length_aspect_ratio', 'scatter_ratio', 'elongatedness', 'pr.axis_rectangularity', 'max.length_rectangularity', 'scaled_variance', 'scaled_variance.1', 'scaled_radius_of_gyration', 'scaled_radius_of_gyration.1', 'skewness_about', 'skewness_about.1', 'skewness_about.2', 'hollows_ratio']
fig, ax = plt.subplots(len(col_names), figsize=(15, 20))
for i, col_val in enumerate(col_names):
sns.distplot(clean_vehicle2[col_val], hist=True, ax=ax[i])
ax[i].set_title('Freq dist '+col_val, fontsize=10)
ax[i].set_xlabel(col_val, fontsize=8)
ax[i].set_ylabel('Count', fontsize=8)
plt.show()
Circuarity:¶
In [78]:
#Understanding Distribution over 'Circularity Attribute'
clean_vehicle2['circularity'].describe().transpose()
Out[78]:
In [107]:
#Plotting Distribution
fig, ax = plt.subplots()
fig.set_size_inches(25, 5)
sns.countplot(x = 'circularity', palette="rocket", data = complete_data)
ax.set_xlabel('circualrity', fontsize=25)
ax.set_ylabel('Count', fontsize=25)
ax.set_title('circularity', fontsize=25)
sns.despine()
In [79]:
#Plotting Dist Plot of Circularity Attribute
fig,ax = plt.subplots()
fig.set_size_inches(30, 12)
sns.distplot(clean_vehicle2['circularity'], hist = True, kde = True,
bins=int(180/5), color = 'blue',
hist_kws={'edgecolor':'black'})
ax.set_xlabel('circularity', fontsize=15)
Out[79]:
In [100]:
sns.FacetGrid(complete_data, hue = 'class', size = 10).map(sns.distplot, 'circularity').add_legend()
Out[100]:
In [ ]:
In [83]:
sns.jointplot(x = "circularity", y = "max.length_rectangularity", data = clean_vehicle2);
In [155]:
f, ax = plt.subplots(figsize = (10, 10))
cmap = sns.cubehelix_palette(as_cmap = True, dark = 0, light = 1, reverse = True)
sns.kdeplot(clean_vehicle2['circularity'], clean_vehicle2['max.length_rectangularity'], cmap = cmap, n_levels = 60, shade=True)
Out[155]:
In [148]:
sns.relplot( x = "circularity", y = "elongatedness", data = clean_vehicle2);
distance_Circularity:¶
In [150]:
clean_vehicle2['distance_circularity'].describe()
Out[150]:
In [151]:
fig, ax = plt.subplots()
fig.set_size_inches(25, 5)
sns.countplot(x = 'distance_circularity', palette="rocket", data = complete_data)
ax.set_xlabel('distance_circualrity', fontsize=25)
ax.set_ylabel('Count', fontsize=25)
ax.set_title('distance_circularity', fontsize=25)
sns.despine()
In [152]:
#Plotting Dist Plot of Age Attribute
fig,ax = plt.subplots()
fig.set_size_inches(30, 12)
sns.distplot(clean_vehicle2['distance_circularity'], hist = True, kde = True,
bins=int(180/5), color = 'blue',
hist_kws={'edgecolor':'black'})
ax.set_xlabel('distance_circularity', fontsize=15)
Out[152]:
In [153]:
sns.FacetGrid(complete_data, hue = 'class', size = 10).map(sns.distplot, 'distance_circularity').add_legend()
Out[153]:
In [ ]:
In [160]:
f = plt.figure()
f, axes = plt.subplots(nrows = 2, ncols = 2, sharex=True, sharey = True)
f.set_size_inches(15, 15)
axes[0][0].scatter(x = "distance_circularity", y = "scaled_variance", data = clean_vehicle2)
axes[0][0].set_xlabel('distance_circularity', labelpad = 5)
axes[0][0].set_ylabel('scaled_variance', labelpad = 5)
axes[0][1].scatter(x = "distance_circularity", y = "scaled_variance.1", data = clean_vehicle2)
axes[0][1].set_xlabel('distance_circularity', labelpad = 5)
axes[0][1].set_ylabel('scaled_variance.1', labelpad = 5)
axes[1][0].scatter(x = "distance_circularity", y = "pr.axis_rectangularity", data = clean_vehicle2)
axes[1][0].set_xlabel('distance_circularity')
axes[1][0].set_ylabel('pr.axis_rectangularity', labelpad = 5)
axes[1][1].scatter(x = "distance_circularity", y = "elongatedness", data = clean_vehicle2)
axes[1][1].set_xlabel('distance_circularity')
axes[1][1].set_ylabel('elongatedness', labelpad = 5)
Out[160]:
Scatter_ratio¶
In [161]:
clean_vehicle2['scatter_ratio'].describe()
Out[161]:
In [163]:
#Plotting Dist Plot of Age Attribute
fig,ax = plt.subplots()
fig.set_size_inches(30, 12)
sns.distplot(clean_vehicle2['scatter_ratio'], hist = True, kde = True,
bins=int(180/5), color = 'blue',
hist_kws={'edgecolor':'black'})
ax.set_xlabel('scatter_ratio', fontsize=15)
Out[163]:
In [177]:
sns.relplot( x = "scatter_ratio", y = "elongatedness", data = clean_vehicle2);
sns.relplot( x = "scatter_ratio", y = "scaled_variance", data = clean_vehicle2);
sns.relplot( x = "scatter_ratio", y = "pr.axis_rectangularity", data = clean_vehicle2);
Elongatedness:¶
In [178]:
clean_vehicle2['elongatedness'].describe()
Out[178]:
In [180]:
fig, ax = plt.subplots()
fig.set_size_inches(25, 5)
sns.countplot(x = 'elongatedness', palette="rocket", data = complete_data)
ax.set_xlabel('elongatedness', fontsize=25)
ax.set_ylabel('Count', fontsize=25)
ax.set_title('elongatedness', fontsize=25)
sns.despine()
In [182]:
#Plotting Dist Plot of Age Attribute
fig,ax = plt.subplots()
fig.set_size_inches(30, 12)
sns.distplot(clean_vehicle2['elongatedness'], hist = True, kde = True,
bins=int(180/5), color = 'blue',
hist_kws={'edgecolor':'black'})
ax.set_xlabel('elongatedness', fontsize=15)
Out[182]:
In [183]:
sns.FacetGrid(complete_data, hue = 'class', size = 10).map(sns.distplot, 'elongatedness').add_legend()
Out[183]:
In [184]:
sns.relplot( x = "elongatedness", y = "scaled_variance.1", data = clean_vehicle2);
sns.relplot( x = "elongatedness", y = "scaled_variance", data = clean_vehicle2);
sns.relplot( x = "elongatedness", y = "pr.axis_rectangularity", data = clean_vehicle2);
pr.axis_rectangularity:¶
In [186]:
clean_vehicle2['pr.axis_rectangularity'].describe()
Out[186]:
In [187]:
fig, ax = plt.subplots()
fig.set_size_inches(25, 5)
sns.countplot(x = 'pr.axis_rectangularity', palette="rocket", data = complete_data)
ax.set_xlabel('pr.axis_rectangularity', fontsize=25)
ax.set_ylabel('Count', fontsize=25)
ax.set_title('pr.axis_rectangularity', fontsize=25)
sns.despine()
In [188]:
#Plotting Dist Plot of Age Attribute
fig,ax = plt.subplots()
fig.set_size_inches(30, 12)
sns.distplot(clean_vehicle2['pr.axis_rectangularity'], hist = True, kde = True,
bins=int(180/5), color = 'blue',
hist_kws={'edgecolor':'black'})
ax.set_xlabel('pr.axis_rectangularity', fontsize=15)
Out[188]:
In [189]:
sns.FacetGrid(complete_data, hue = 'class', size = 10).map(sns.distplot, 'pr.axis_rectangularity').add_legend()
Out[189]:
In [190]:
sns.relplot( x = "pr.axis_rectangularity", y = "scaled_variance", data = clean_vehicle2);
sns.relplot( x = "pr.axis_rectangularity", y = "scaled_variance.1", data = clean_vehicle2);
In [213]:
from scipy.stats import zscore
XScaled = clean_vehicle2.apply(zscore)
XScaled.head()
Out[213]:
Finding Top 12 Features: Features Selection¶
Here we are having 18 features and 846 records. We are likely to have the Curse of Dimensionality.
Statistical tests can be used to select those features that have the strongest relationship with the output variable.
The scikit-learn library provides the "SelectKBest class" that can be used with a suite of different statistical tests to select a specific number of features.
So, I am using chi-squared (chi²) statistical test to select 15 of the best features from the Dataset.
In [212]:
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
X = clean_vehicle2 #independent columns
y = vehicle_df["class"] #target column
#apply SelectKBest class to extract top 12 best features
bestfeatures = SelectKBest(score_func = chi2, k = 12)
fit = bestfeatures.fit(X,y)
dfscores = pd.DataFrame(fit.scores_)
dfcolumns = pd.DataFrame(X.columns)
#concat two dataframes for better visualization
featureScores = pd.concat([dfcolumns,dfscores],axis=1)
featureScores.columns = ['Specs','Score'] #naming the dataframe columns
print(featureScores.nlargest(12,'Score')) #print 12 best features
In [214]:
new_vehicle = clean_vehicle2.drop(columns =['circularity', 'pr.axis_aspect_ratio', 'pr.axis_rectangularity', 'max.length_rectangularity', 'skewness_about.2', 'hollows_ratio'])
XScaled_new = new_vehicle.apply(zscore)
XScaled_new.head()
Out[214]:
In [64]:
#Visually inspect the covariance between independent dimensions
sns.pairplot(XScaled, diag_kind='kde')
Out[64]:
In [215]:
from sklearn.decomposition import PCA
In [216]:
pca = PCA().fit(XScaled)
#Plotting the Cumulative Summation of the Explained Variance
plt.figure(figsize=(10, 10))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.title('Pulsar Dataset Explained Variance')
plt.show()
This plot tells us that selecting 7 components we can preserve something around 96% of the total variance of the data.¶
It makes sense, we’ll not use 100% of our variance, because it denotes all components, and we want only the principal ones.¶
With this information in our hands, we can implement the PCA for 7 best components by using this snippet of code:¶
In [236]:
pca = PCA(n_components = 7, random_state = 1)
dataset = pca.fit_transform(XScaled)
In [237]:
print(pca.explained_variance_)
In [238]:
print(pca.components_)
print('-----------------------------------------------------------------')
print(pca.explained_variance_ratio_)
In [239]:
Xpca7 = pca.transform(XScaled)
print(Xpca7)
In [71]:
sns.pairplot(pd.DataFrame(Xpca12))
Out[71]:
Constructing SVC Model with Feature Selection¶
In [223]:
X = XScaled_new
y = vehicle_df["class"]
svc_clf = SVC()
svc_clf.fit(X, y)
svc_score = svc_clf.score(X, y)
print('Overall Acuracy: ', svc_score)
Constructing SVC Model with Dimensionality Reduction Technique: PCA¶
In [240]:
X1 = Xpca7
y = vehicle_df["class"]
svc_clf = SVC()
svc_clf.fit(X1, y)
svc_score = svc_clf.score(X1, y)
print('Overall Acuracy: ', svc_score)
Thus by implementing PCA, accuracy of our model increased by almost 4% significantly.¶
Implementing Optimal Grid Search and hypertuning Our SVC model¶
In [225]:
from sklearn.model_selection import GridSearchCV
In [241]:
param_grid = {'C':[0.01, 0.05, 0.5, 1], 'kernel':['linear','rbf']}
grid = GridSearchCV(SVC(), param_grid, refit = True, verbose = 2)
# Fitting the Tuned Model
grid.fit(X1, y)
grid_score = grid.score(X1, y)
print('Overall Acuracy: ', grid_score)
Accuracy obtained by Optimal Grid Search is almost same as obtained by PCA SVC Model.¶
In [242]:
grid_scores = cross_val_score(grid, X1, y, cv = 3)
print('Cross Val Score: ', grid_scores.mean())
Thus, implementing PCA betters the Performance of the Model. It is because:¶
- PCA Removes Correlated Features: In a real world scenario, this is very common that we got 18 features in our dataset. We cannot run algorithm on all the features as it will reduce the performance of the algorithm and it will not be easy to visualize that many features in any kind of graph. So, we MUST reduce the number of features in our dataset.
After implementing the PCA on your dataset, all the Principal Components are independent of one another. There is no correlation among them.
- PCA Improves Algorithm Performance: With so many features, the performance of the algorithm will drastically degrade. So, PCA is used to speed up Machine Learning algorithm by getting rid of correlated variables which don't contribute in any decision making. The processing time of the algorithms reduces significantly with less number of features.
So, if the input dimensions are too high, use of PCA speeds up the algorithm.
PCA Reduces Overfitting: Overfitting mainly occurs when there are too many variables in the dataset. So, PCA helps in overcoming the overfitting issue by reducing the number of features.
PCA Improves Visualization: It is very hard to visualize and understand the data in high dimensions. PCA transforms a high dimensional data to low dimensional data so that it can be visualized easily.
In [ ]: